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5.1.5 Conditional Expectation (Revisited) and Conditional 
Variance 

In Section 5.1.3 . we briefly discussed conditional expectation. Here, we will discuss the 
properties of conditional expectation in more detail as they are quite useful in practice. We 
will also discuss conditional variance. An important concept here is that we interpret the 
conditional expectation as a random variable. 

Conditional Expectation as a Function of a Random 
Variable: 

Remember that the conditional expectation of X given that Y = y is given by 

E[X\ Y = y]= Yj x i P X | y( x i I y)- 

x i ER X 

Note that E[X\ Y = v] depends on the value of y. In other words, by changing y, E[X \ Y = y] 
can also change. Thus, we can say E\X\ Y = v] is a function of v, so let's write 

g(y) = E[X\Y = y]. 

Thus, we can think of g(y) = E\X\ Y = v] as a function of the value of random variable Y. We 
then write 


g(Y) = E[X | 7]. 

We use this notation to indicate that E[X\ Y] is a random variable whose value equals 
g(y) = E[X | Y = y] when Y = y. Thus, if Tis a random variable with range R y = {v 1 ,y 2 , •"}, 

then E\X\ Y] is also a random variable with 

/ E[X | Y= Vy] with probability P(Y = Vj) 

E[X | Y = v 2 ] with probability P(Y = v 2 ) 

E[X\ V] = 1 
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Let's look at an example. 


Example 5.10 LetX = aY + b. Then E[X\ Y = y\ = E[ciY + b \ Y = y\ = ay + b. Here, we have 
g(y) = ay + b, and therefore, 


E[X | Y] = aY+b, 

which is a function of the random variable Y. 


Since E[X\ Y] is a random variable, we can find its PMF, CDF, variance, etc. Let's look at an 
example to better understand E[X\ Y]. 

Example 5.11 Consider two random variables X and Y with joint PMF given in Table 5.2. Let 
Z = E[X | 7], 

a. Find the Marginal PMFs ofXand Y. 

b. Find the conditional PMF of X given 7=0 and 7=1, i.e., find P x | y(x | 0) and 

p x\y( x I !)• 

c. Find the PMF of Z. 

d. Find EZ, and check that EZ = EX. 

e. Find Var(Z). 


Table 5.2: Joint PMF of X and Y in example 5.11 



o 

II 

7= 1 


1 

2 

c 

II 

* 

— 

— 


5 

5 


2 


X= 1 

— 

0 


5 



• Solution 

o a. Using the table we find out 
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^0) 

Pjf?) 

P y(0) 

Pyil) 


1 

5 

2 

5 

1 

5 

2 

5 



3 

5’ 


+ 0 = 


2 

5’ 


+ 


2 3 

5 ~ 5’ 



2 

Thus, the marginal distributions of X and Y are both Bernoulli (j). However, 

note that X and Y are not independent, 
b. We have 


Thus, 


Px\ y(010) 

l 

5 1 

= ! = 3' 

5 


Pxy( 0, 0) 
Py(0) 




We conclude 


X\Y= 0 


Bernoulli 


Similarly, we find 

^|y(0| D = ^ 

p x |y(l 11) = 0. 


Thus, given Y = 1, we have always X = 0. 
c. We note that the random variable Y can take two values: 0 and 1. Thus, the 
random variable Z = E[X\ Y] can take two values as it is a function of Y. 
Specifically, 
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( E[X\Y=0\ if 7=0 

Z=E[X\Y]= | 

U[X|7=1] if 7= 1 

Now, using the previous part, we have 

2 

E[X\Y=0]= -, E[X\Y= 1] = 0, 

3 2 

and since P(y = 0) = j, and P(y = 1) = j, we conclude that 


/ 2 3 

- with probability ~ 

Z = E[X\Y} = I 

2 

0 with probability ~ 


So we can write 



if z = 


2 

3 


= < \ if z = 0 


' 0 otherwise 


d. Now that we have found the PMF of Z, we can find its mean and variance. 
Specifically, 


3 2 2 


2 

We also note that EX = -. Thus, here we have 

E[X | = E[Z\ = E[E[X | 7]]. 

In fact, as we will prove shortly, the above equality always holds. It is 
called the law of iterated expectations, 
e. To find Var(Z), we write 
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where 


Thus, 


Var(Z) = E[Z 2 ] 
= E[Z 2 ] 


(EZ) 2 

4 

25’ 


E[Z 2 ] 


4 

9 



4 

15' 


4 4 

v ar(Z) - - - - 

8 

~ 75' 


Example 5.12 

Let X and The two random variables and g and h be two functions. Show that 

E\g(X)h(Y)\X\=g{X)E[h{Y)\X\. 


• Solution 

o Note that E[g(X)h( Y) \ X] is a random variable that is a function of X. In particular, 
if X = x, then E\g(X)h( Y) X] = E\g(X)h( Y) X = x]. Now, we can write 

E[g(X)h(Y)\X=x] = E\g(x)h(Y) \X = x] 

= g(x)E[h(Y) \X= x] (since g(x) is a constant). 

Thinking of this as a function of the random variable X, it can be rewritten as 
E\g(X)h(Y) | X\ = g(X)E[h(Y)\X\. This rule is sometimes called "taking out what 
is known." The idea is that, given X, g(X) is a known quantity, so it can be taken 
out of the conditional expectation. 


E[g(X)h(Y)\X\ = g(X)E[h(Y)\X\ (5.6) 
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Iterated Expectations: 

Let us look again at the law of total probability for expectation. Assuming g(Y) = E\X\ Y], we 
have 

E[X\ = X E[X\Y = yj ]Py(yj) 

yj eR Y 

= Z siypPyiyj ) 

VjERy 

= E[g(Y)\ by LOTUS (Equation 5.2) 

= E[E[X | Y]]. 


Thus, we conclude 


E[X] = E[E[X | Y]]. (5.7) 

This equation might look a little confusing at first, but it is just another way of writing the law 
of total expectation (Equation 5.4). To better understand it, let's solve Example 5.7 using this 
tenninology. In that example, we want to find EX. We can write 

E[X] =E[E[X\N]\ 

= E[Np] (since A| /V ~ Binomial(N,p )) 

= pE[N] = pX. 

Equation 5.7 is called the law of iterated expectations. Since it is basically the same as 
Equation 5.4, it is also called the law of total expectation [3]. 


Law of Iterated Expectations: E[X\ = E[E[X\ 7]] 


Expectation for Independent Random Variables: 

Note that if two random variables X and Y are independent, then the conditional PMF of X 
given Y will be the same as the marginal PMF of X, i.e., for any x E R x , we have 

p X \ )P X It) = P / X P 

Thus, for independent random variables, we have 
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E[X\ Y = y] = Yj xP x | ylx | y) 

xE R x 

= Y xP j/ x ) 

xE R x 

= E[X\. 


Again, thinking of this as a random variable depending on Y, we obtain 

E\X | 7] = E\X\, when X and Y are independent. 

More generally, if X and Y are independent then any function of X, say g(X), and Y are 
independent, thus 


E[g(X)\Y]=E[g(X)]. 

Remember that for independent random variables, P XY (x,y) = P > i.x)P y(y). From this, we can 
show that E[XY] = EXEY. 


Lemma 5.2 

If A and Y are independent, then E[XY\ = EXEY. Using LOTUS, we have 


E[XY] 


I I x yP xy( x ’ y) 

xER x yER v 

I I xyP y(x)P yiy) 

xER x yER y 



EXEY. 


Note that the converse is not true. That is, if the only thing that we know about X and Y is that 
E\XY] = EXEY, then A and Y may or may not be independent. Using essentially the same 
proof as above, we can show if A and Y are independent, then E[g(X)h( 7)] = E[g(X)]E[li(Y)] 
for any functions g: R ■-» R and /? : R R. 


If A and 7 are independent random variables, then 

1. E[X\ 7] = EX; 

2. E[g(X)\Y] = E[g(X)]; 

3. E[XY] = EXEY; 

4. E[g(X)h{Y)\=E\g(X)\E[h{Y)l 
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Conditional Variance: 


Similar to the conditional expectation, we can define the conditional variance of X, Var 
(X\ Y = y ), which is the variance of X in the conditional space where we know Y = y. If we let 
Bx\ yO’) = E \. x \ Y = y L then 

Var(X| Y = y) = E[(X-p x] yfy)) 2 \Y = y] 

= X ( x i ~ t i x\ &)Y p x\ y( x /) 

x i G R X 

= E[X 2 \Y = y]-f, xlY iv) 2 . 

Note that Var(X| Y = y) is a function of y. Similar to our discussion on E[X\ Y = y] and E\X\ Y] 
, we define Var(X| Y) as a function of the random variable Y. That is, Var(X| Y) is a random 
variable whose value equals Var(X| Y = v) whenever Y = y. Let us look at an example. 


Example 5.13 

Let X, Y, and Z = E[X\ Y] be as in Example 5.11. Let also V = Var(X| 7). 


a. Find the PMF of V. 

b. Find EV. 

c. Check that Var(X) = E(V) + Var(Z). 


• Solution 


2 

In Example 5.11, we found out that X, Y ~ BernouUi(-). We also obtained 


X\Y= 0 


Bernoulli 


(!> 


P(X= 0| 7= 1) = 1, 

Var(Z> - Ty 

a. To find the PMF of V, we note that Fisa function of Y. Specifically, 

,Var(X| 7=0) if 7=0 


V = Var(X| 7) 


.Var(X| 7=1) if 7=1 


Therefore, 
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/ 3 
Var(X| 7=0) with probability - 

V = Var(X| 7) = / 

2 

Var(X| 7=1) with probability - 


Now, since X\ 7 = 0 ~ Bernoulli | j j, we have 


2 1 2 
Var(X| 7=0)= -•- = -, 


and since given 7 = 1, X = 0, we have 


Var(X| 7= 1) = 0. 

Thus, 


I 2 3 

g with probability - 

V = Var(X| 7) = / 

2 

0 with probability - 


So we can write 


3 

5 

PM = 2 - 


2 

ifV = g 

if v = 0 


b. To find EV, we write 



V 0 otherwise 


3 2 2 
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c. To check that Var(V) = E(V) + Var(Z), we just note that 


2 3 

Var W 



Var(Z) = 


8 

75' 


6 

25’ 


In the above example, we checked that Var(V) = E(V) + Var(Z), which says 

Var(V) = £(Var(X| 7)) + Var(£[V| 7]). 

It turns out this is true in general and it is called the law of total variance, or variance 
decomposition formula [3]. Let us first prove the law of total variance, and then we explain it 
intuitively. Note that if V = Var(X| 7), and Z = E[X 7], then 

V = E[X 2 | 7] - (E[X\ 7]) 2 
= E[X 2 | 7] - Z 2 . 


Thus, 

EV = E[E[X 2 | 7]] - E[Z 2 ] 

= E[X 2 ~\ - E[Z 2 ] (law of iterated expectations(Equation 5.7)) (5.8) 

Next, we have 

Var(Z) = E[Z 2 ] - (EZ) 2 

= E[Z 2 \ - (EX) 2 (law of iterated expectations) (5.9) 
Combining Equations 5.8 and 5.9, we obtain the law of total variance. 


Law of Total Variance: 

Var(V) = /:[Var(V 7)] + Var(C[V| 7]) (5.10) 
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There are several ways that we can look at the law of total variance to get some intuition. Let 
us first note that all the terms in Equation 5.10 are positive (since variance is always positive). 
Thus, we conclude 


Var(V) > E(Vav(X\ 7)) (5.11) 

This states that when we condition on Y, the variance of X reduces on average. To describe this 
intuitively, we can say that variance of a random variable is a measure of our uncertainty about 
that random variable. For example, if Var(V) = 0, we do not have any uncertainty about X. 
Now, the above inequality simply states that if we obtain some extra information, i.e., we 
know the value of Y, our uncertainty about the value of the random variable X reduces on 
average. So, the above inequality makes sense. Now, how do we explain the whole law of total 
variance? 

To describe the law of total variance intuitively, it is often useful to look at a population 
divided into several groups. In particular, suppose that we have this random experiment: We 
pick a person in the world at random and look at his/her height. Let's call the resulting value X. 
Define another random variable Y whose value depends on the country of the chosen person, 
where Y = 1, 2, 3,... , n, and n is the number of countries in the world. Then, let's look at the 
two terms in the law of total variance. 

Var(V) = E(y.av(X\ Y)) + Vav(E[X\ 7]). 

Note that Var(X| Y = i ) is the variance of X in country i. Thus, E(Wax(X\ 7)) is the average of 
variances in each country. On the other hand, E[X\ Y = /'] is the average height in country i. 
Thus, Var(/:[V 7]) is the variance between countries. So, we can interpret the law of total 
variance in the following way. Variance of V can be decomposed into two parts: the first is the 
average of variances in each individual country, while the second is the variance between 
height averages in each country. 

Example 5.14 

Let N be the number of customers that visit a certain store in a given day. Suppose that we 
know fi[/V] and Var(/V). Let V ; be the amount that the z'th customer spends on average. We 

assume X-s are independent of each other and also independent of N. We further assume they 
have the same mean and variance 

EX i = EX, 

Var(V ( ) = Var(V). 

Let 7be the store's total sales, i.e., 


N 


y- 2X 

i= 1 
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Find EY and Var(7). 


• Solution 

o To find EY, we cannot directly use the linearity of expectation because TV is 

random. But, conditioned on TV = n, we can use linearity and find E[Y\ TV = n]; so, 
we use the law of iterated expectations: 

EY = E[E[Y\ A/]] (law of iterated expectations) 


N 


= E 



TV 


N 


= E 


i= 1 


(linearity of expectation) 


N 


= E 


i= 1 


(X-s and TV are indpendent) 


= E[NE[X |] (since EX j = EX: s) 

= E[X\E[N] (since EX is not random). 


To find Var(7), we use the law of total variance: 

Var(7) = £(Var(7|TV)) + Var(£[T| ,V]) 

= E(Va.r(Y\ TV)) + Var (NEX) (as above) 

= £(Var( 71 TV)) + (EX) 2 Xav(N) (5.12) 

To find E(Var(Y\N)), note that, given TV = n, 7 is a sum n independent random 
variables. As we discussed before, for n independent random variables, the 
variance of the sum is equal to sum of the variances. This fact is officially proved 
in Section 5.3 and also in Chapter 6, but we have occasionally used it as it 
simplifies the analysis. Thus, we can write 

N 

Var(7|TV) = X Var ^ ; I A 0 
/ = 1 
N 

= X Var (Xj) (since X-’s are independent of TV) 

i=l 

= NVar(X). 


Thus, we have 
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E(Vi ar( Y \ N)) = ENVar{X) (5.13) 

Combining Equations 5.12 and 5.13, we obtain 

Var(7) = ENVar(X) + (EX) 2 Var(N). 


<— previous 
next —> 
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